Persist PSF products and add the two campaign-level merges (exp_persist, star_cat_merge, final_cat_merge) - #879

One rule per exposure, output = ONE manifest on the persistent root at
<products_dir>/exp/<shard>/<exp>/manifests/exp_persist.json. Not a directory()
output: what we want written down is which files were copied and how big each
was, and a directory attests only that a directory exists. Byte-stable, so a
no-op rerun does not move an mtime clean_exposure reads.

Its own rule rather than a cp on the end of exp_psf, and that is the whole
point: the keep list rides on params, so adding a pattern reruns seconds of
copying instead of four hours of PSF fitting per exposure.

A localrule, by the arithmetic that made exp_star_cat one -- a few MB of cp,
~20k of them at DR6 scale, each shorter than the scheduling latency that would
submit it. The mid-chain grouping constraint does not bite: its neighbours are
exp_psf (too heavy to fuse) and clean_exposure (local already).

clean_exposure gains the manifest as an input, so a store is never reclaimed
before its keepers have left scratch -- conditional only on there being a keep
list, since "keep nothing" must not become a dependency on a rule that would
fail for having nothing to copy.

rule all requests the persist manifests DIRECTLY, not only through
clean_exposure: the purge takes the store whether or not clean: is on, so
hanging the copy off reclamation alone would lose everything in a clean:false
campaign. Cleaned exposures are excluded -- their exp_psf manifest is gone, so
asking would rebuild the chain from VOS, and a tombstone already means the copy
happened.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

… omits it

run_report disk-scans the scratch run_dir, and exp_persist's manifest is the one
exposure manifest that lives on products_dir instead -- the placement that makes
it survive clean_exposure. Listed in EXP_STAGES it would read as "not run" for
every exposure in the campaign, so it is deliberately absent, with the reason on
the line.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

…e copies

Inodes, not bytes, bind on /project (~1 M-file group quota): smk-m2 measured
~200 loose files per exposure with all candidates on — 25k for 64 tiles, ~2 M
at DR6 scale, for 7 GB. persist_exp.py now writes <products_dir>/exp/<shard>/
<exp>/psf/<exp>.tar (uncompressed, flat members, deterministic: ownership
zeroed, sorted, tmp-cmp-mv so a no-op rerun keeps the mtime) and the manifest
lists every member. Manifest path, rule wiring and params are unchanged.

config.yaml's candidate table carries the smk-m2 per-exposure sizes;
psfex_cat and star_stat are marked unmeasured (no live store held them).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015qtLUV3bVLPV5p6un7aTFR

An orphaned tar.tmp on /project is an inode nothing revisits — the leak the
tar design exists to avoid. try/finally around both tmp writes.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015qtLUV3bVLPV5p6un7aTFR

develop no longer has exp_star_cat / star_catalogue (PR #847); the three
comments that cited exp_star_cat as the localrule precedent now cite
clean_exposure, which makes the same argument.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbnPCyzuDNTgkg715pHhar

PERSISTENCE KEYED ON THE WRONG FILE, and the consequence was the avalanche it
exists to prevent. persist_targets() skipped an exposure only when its SCRATCH
tombstone was there, but clean_exposure is one of two ways a store disappears
and the other leaves nothing behind: /scratch is purged on a 60-day window
whether or not the workflow reclaimed anything. After a purge — or on any
campaign with clean: false — every exposure looked live, its persist manifest
was requested, its exp_psf manifest was gone, and snakemake rebuilt the whole
exposure chain from VOS.

The test is now exp_store_reclaimed(), used by persist_manifests() and by
star_cat_merge's live/reclaimed split so the two cannot disagree, and it takes
BOTH pieces of evidence that an exposure once had a store: a tombstone, or a
persisted manifest with no exp_psf manifest beside it. Both are needed. The
manifest clause alone regressed the tombstone case — measured on smk-g6, whose
126 exposures were cleaned before exp_persist existed and so have no manifest:
the dry run grew 126 exp_get_images, exp_split, exp_psf and exp_persist jobs,
a whole campaign rebuilt from VOS. The tombstone clause alone is the purge bug
above. Neither file can be dropped, because the parse cannot otherwise tell a
store that is GONE from one not BUILT yet, and a fresh campaign must still be
asked to persist.

OVERLAPPING KEEP PATTERNS FAILED EVERY EXPOSURE. persist_exp treated a file
matched by two patterns as a flat-member name collision, so
'validation_psf-*.fits' alongside '*.fits' — an ordinary way to write a keep
list — aborted the pack. Two DIFFERENT paths on one member name is still
fatal; the same path twice is now one file, recorded under the first pattern
that matched it.

merge_star_cat MATERIALISED THE WHOLE CAMPAIGN before merging a row: every
member's bytes, ~2 MB per exposure, ~40 GB at DR6's ~20k exposures against a
rule asking for 16 GB. TarMembers hands the merge class the same entries one
tar at a time, so peak memory is one member plus the class's own accumulators,
which are the unavoidable term. It keeps __len__ off the manifests so the
count is still logged before a tar is opened.

THE STAR MERGE RERAN ON BOOKKEEPING. Its fingerprint was over input PATHS, and
an exposure's edge flips from its manifest to its tar the moment its store is
reclaimed — so every reclamation pass reran the merge over identical content.
It is over the exposure IDS now, which move only when the set does, and which
are what the job derives on its own side. final_cat_merge's is over tile ids
for the same reason.

merge_final_cat's MISSING-COLUMN CHECK WAS UNREACHABLE. create_final_cat's
read_data wraps its column selection in a bare `except:` that prints and falls
through, so a missing column left its return values unbound and the caller got
UnboundLocalError from the return statement, naming nothing. The columns are
checked against the catalogue's own header before read_data is called, and the
message now names every missing one.

A TILE LISTED TWICE killed the merge on the second create_dataset. The tile
list is appended to by hand, so duplicates happen; campaign_tiles() dedupes it
order-preserving, and the Snakefile's TILES does the same so the fingerprint
and the job's derived set still name the same set.

merge_class OFFERED MCCD AND SETOOLS while only MergeStarCatPSFEX had learned
the [fileobj, name] entry shape. Both now take the entry's name from its last
element like PSFEX does — unchanged for the module runner, whose entries are
[path]. Setools needs one thing more before it can read a tar (it hands
file_io input_file_list[0][0] as a template path), and merge_class says so
rather than implying otherwise.

Also: README no longer lists scripts/sp_rule.py, which does not exist.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbnPCyzuDNTgkg715pHhar

Three defects in scripts/python/create_final_cat.py, fixed where they live
rather than worked around in the workflow rule that now calls it. A hand-run
of the tool deserves them as much as the rule does, and files it has already
written carry the first one.

copy_data allocated np.empty with the SOURCE catalogue's full dtype and then
filled only the requested columns, so every column NOT in the parameter file
reached the hdf5 file as uninitialised memory: meaningless values, and
different bytes on every run over the same inputs. It allocates the requested
columns alone now, in the source catalogue's order. The parameter file says
what the merged catalogue is for; those are the columns it gets.

read_param_file returned list(set(...)), whose order varies with the process's
string hash seed. Column order is part of a structured dtype and therefore
part of the file, so two runs over the same inputs disagreed. Ordered dedup
via dict.fromkeys. (The duplicate-count message also only printed for more
than one duplicate, and said {n} literally.)

read_data wrapped its column selection in a bare `except:` that printed and
fell through, leaving its return values unbound — so a missing column surfaced
to the caller as UnboundLocalError from the return statement, naming neither
the file nor the column. It raises a KeyError naming the file and every
missing column, in parameter-file order.

process()'s own create_dataset follows the array copy_data returns rather than
the source dtype, which are no longer the same thing.

merge_final_cat.py drops the equivalents it had been carrying at the call site
and relies on the fixed functions. The fixture hdf5 is byte-identical either
way (md5 6de2d261…): the workaround and the fix produce the same file, which
is the point.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbnPCyzuDNTgkg715pHhar

Both merges had a constant mem_mb, which is wrong by however much a campaign
differs from the one it was tuned on — and these are the only two rules whose
single job grows with the whole campaign. Both are now measured slopes,
evaluated against the campaign's own bytes at DAG build, still * attempt.

STAR SIDE, measured on this login node in the campaign container over
synthetic tars, 20 and 80 exposures of 40 CCDs x 400 stars:

    input members      peak RSS (getrusage RUSAGE_CHILDREN)
    32.3 MB            383 MB
    129.0 MB          1313 MB

a slope of 10.1x input bytes over a ~73 MB interpreter floor. Tenfold because
MergeStarCatPSFEX accumulates every column into python LISTS of python floats
before building the output arrays. THE CONSEQUENCE IS A CEILING and the
Snakefile says so: at ~2 MB of members per exposure a 16 GB job merges roughly
800 exposures, and DR6's ~20k would want ~400 GB. A full-survey full_starcat
needs that accumulation changed to preallocated arrays or a two-pass count —
a change to MergeStarCatPSFEX, not to this rule, and not in this PR. The
formula is honest about the slope so the job asks for what it will use and
fails at submission rather than most of the way through.

TILE SIDE, measured against smk-g6's real catalogues, 2 tiles (73.9 MB in,
largest 39.6 MB) and 6 tiles (235.5 MB in, largest 47.7 MB): peak RSS 129 MB
and 139 MB. FLAT in the tile count, because the merge holds one catalogue at a
time — so it is sized on the LARGEST tile at ~3x, not on the total. Runtime is
the total, since every tile is read end to end. On smk-g6's 64 tiles the rule
resolves to mem_mb=1002, runtime=51, against the flat 8000/120 it had.

Sizes come from stat() on the tar or the catalogue, falling back to the
measured per-unit default when a fresh campaign has not produced it yet.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbnPCyzuDNTgkg715pHhar

Closes the readability half of #844, and makes the
2026-09-08 call's request — keep the PSF model — something you can write down
as `psf_model` rather than `*.psf`.

`persist_exp:` entries are now names from a catalogue in persist_exp.py, which
is the single source of truth for what each one means, what it costs per
exposure and what keeping it buys:

    psf_validation  validation_psf-*.fits        2.0 MB
    psf_model       *.psf                        2.8 MB
    psfex_cat       psfex_cat-*.cat            unmeasured
    star_selection  star_selection-*.fits       24.5 MB
    star_train      star_split_ratio_80-*.fits  19.9 MB
    star_test       star_split_ratio_20-*.fits   7.1 MB
    star_stats      star_stat-*.txt            unmeasured

`persist_exp.py --list-products` renders it, and config.yaml's block IS that
rendering rather than a second copy of it — the old block was a long comment
listing globs and their sizes, maintained by hand beside the code that
actually knew them.

THE DEFAULT BECOMES psf_validation + psf_model, ~4.8 MB per exposure. The
model is the single most capability-adding thing an exposure can keep: with
it the PSF can be re-interpolated at any position later without rebuilding the
chain from VOS, and without it that capability dies with the /scratch purge.

A raw glob is still accepted as an escape hatch for a file the catalogue does
not name yet. The test is syntactic and cheap — a glob metacharacter or a dot
means glob, a bare identifier means name — so `*.psf` and `psf_model` cannot
be confused. An unknown NAME is a parse-time WorkflowError listing the valid
ones, not a silently empty keep or a per-exposure failure an hour in.

The manifest records both: `products` as written, `patterns` resolved, and
each member's own `product`. star_cat_merge's gate and its member glob resolve
through the same catalogue, so adding a product cannot leave the two
disagreeing, and the "add this to persist_exp" hint now names the product.

Tile-side retention is explicitly out of scope and noted as such in
config.yaml: final_cat is the only tile product that persists today, and it is
written straight to products_dir by tile_make_cat.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbnPCyzuDNTgkg715pHhar

All three merge classes built every output column by extending a python list
with one value per star: `x += list(data["X"])`. Four bytes of float32 payload
became a 32-byte python object plus an 8-byte pointer in an overallocating
list, measured end to end at ~10x the input bytes — which put a full-survey
full_starcat (~20k exposures x 40 CCDs) at ~400 GB of RAM and out of reach of
any node.

One array per input catalogue per column, concatenated once at the end. Same
values, same order, same dtypes — np.array() over a list of numpy scalars and
np.concatenate() over the arrays they came from agree on both. The stacking
helper empties the list it is handed, which is half the saving: concatenate
holds the chunks and the result at once, so releasing column by column peaks
at one campaign plus one column rather than two campaigns.

MEASURED on the same two fixture points, 20 and 80 exposures of 40 CCDs x 400
stars:

    input members    peak RSS, before    after
    32.3 MB          383 MB              238 MB
    129.0 MB        1313 MB              740 MB

10.1x -> 5.5x, over a ~62 MB interpreter floor. The rule's mem_mb factor
follows. A 16 GB job now merges ~2200 exposures rather than ~800.

THE REMAINING 5.5x IS THE OUTPUT SIDE: file_io writes every float column as
FITS 1D, so float32 inputs become a float64 table astropy then buffers. That
is a change to the output FORMAT, which is what sp_validation reads, and a
different decision from this one.

BYTE-IDENTICAL OUTPUT, both ways in. The workflow's tar path and the module
runner's plain [path] path produce the same file as before the change, md5
f7caa1cf… on the fixture — the runner path checked by calling
MergeStarCatPSFEX directly with [[path]] entries as merge_starcat_runner
builds them.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbnPCyzuDNTgkg715pHhar

`persist_exp:` was doing two jobs. It decided what a campaign keeps for later
— a retention question, and the user's — and it also decided whether the
campaign's star catalogue could be built at all, because star_cat_merge
existed only when the keep list happened to name something
validation_psf-shaped. That made the survey's PSF diagnostics an opt-in, and a
typo away from silently absent.

exp_persist now packs psf_validation for every exposure whatever the config
says. It is the merged catalogue's PROVENANCE — a full_starcat with no
per-exposure inputs beside it cannot be audited, re-cut or recomputed after a
purge — and it is what keeps APPENDING TILES CHEAP, since a tile added next
month brings exposures whose catalogues must join the existing stack and the
alternative is rebuilding their chains from VOS. ~2 MB per exposure: ~40 GB
and ~40k inodes at DR6 scale against a ~1 M-inode group quota, which is the
price of being able to say where the number came from.

`persist_exp:` is therefore purely additive retention, defaulting to
psf_model, and an EMPTY list is now a coherent instruction rather than a
switch that turns persistence off: the tar holds the merge's inputs and
nothing else. The keep-list gate on star_cat_merge and its parse-time warning
are gone with it, as is clean_exposure's conditional edge on exp_persist —
there is no configuration left under which that rule has nothing to wait for.

No transient/cleanup knob: these files are kept, not staged.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbnPCyzuDNTgkg715pHhar

…t all

final_cat_merge on smk-g6's real catalogues failed on three of the 67 columns
the parameter file asks for. Two of them the file should not have been asking
for.

IMAFLAGS_ISO is DROPPED. The tile-side SExtractor runs with FLAG_IMAGE = False
and DOT_PARAM_FILE = default_noimaflags.param (config_tile_Sx.ini), so the
column is never written into a tile catalogue and asking for it could only
fail. Instrument flags reach the pipeline on the EXPOSURE side, where
exp_split delivers the flag image and SExtractor reads it.

NGMIX_MOM_FAIL is RENAMED to NGMIX_MCAL_TYPES_FAIL, which is what f0fca23
called it in June and what the catalogues carry.

NGMIX_NEIGHBOUR_FLAG STAYS. It was added to make_cat in fa6e001 on
2026-07-12 and it is the blend flag the systematics tests need. smk-g6's
catalogues do not have it — checked on the files — so final_cat_merge still
fails there, and that failure is correct: the campaign's catalogues are
missing a column the analysis wants, which is a fact about the data and not
about this file. Its launch snapshot is gone (only .snakemake survives under
smk-g6-state), so the run's HEAD cannot be read back; what remains is that its
catalogues carry NGMIX_MCAL_TYPES_FAIL (June) but not NGMIX_NEIGHBOUR_FLAG
(July), consistent with a snapshot taken between the two.

NO MASK COLUMN REPLACES IMAFLAGS_ISO, AND THE FILE NOW SAYS WHY. The intended
replacement is make_cat's per-band MASK_<band>, queried from the healsparse
maps named by MASK_EXT_PATHS — and the workflow sets none: config_tile_Mc.ini
has no such entry, save_mask_ext_data is never called, no MASK_<band> column
exists in any catalogue this workflow has produced, and smk-g6's carry none.
Naming one here would fail every merge on every campaign. The merged catalogue
therefore carries no mask information today; that is a CONFIG gap, and closing
it is setting MASK_EXT_PATHS first and adding the column names second. No
healsparse map is staged under /project/def-mjhudson yet.

With NGMIX_NEIGHBOUR_FLAG set aside, the merge runs clean over all 64 of
smk-g6's real catalogues: 2.50 GB read in 20 s at 151 MB peak RSS, producing a
0.94 GB hdf5 of 65 columns. That also confirms the tile-side sizing — the rule
asks for 1002 MB and 51 minutes.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbnPCyzuDNTgkg715pHhar

The array accumulation landed a commit ago took the star merge from ~10x the
input bytes to ~5.5x. What was left was the accumulation itself: one array per
input catalogue, then a concatenate that has to hold its inputs and its result
at the same time.

MergeStarCatPSFEX now makes two passes. The first reads only the FITS HEADER
of every input — NAXIS2, the row count — and touches no data block; the second
allocates each output column once, at its exact final length, and fills it
slice by slice. There are no chunks and no concatenate, so peak memory is one
output plus one input catalogue.

The workflow's tar reader hands over the archive's own file object rather than
a BytesIO of the whole member, so the counting pass costs a header rather than
a member. Both it and a plain list of paths are iterable twice, which the two
passes require; a one-shot iterable would fill nothing on the second pass, so
the merge checks that the passes agree on the row count rather than writing a
catalogue padded with uninitialised memory.

MEASURED on the same two fixture points, 20 and 80 exposures of 40 CCDs x 400
stars:

    input members   python lists   arrays+concat   two passes
    32.3 MB          383 MB          238 MB          221 MB
    129.0 MB        1313 MB          740 MB          661 MB
    slope             10.1x           5.5x            4.8x

The rule's mem_mb factor follows. A 16 GB job now merges ~1300 exposures.

THE REMAINING 4.8x IS THE OUTPUT SIDE: file_io writes every float column as
FITS 1D, so float32 inputs become a float64 table astropy then buffers — 141 MB
of table for 78 MB of payload at the 80-exposure point. What stands between
here and a full-survey full_starcat is that format, not the merge.

MergeStarCatMCCD and MergeStarCatSetools keep the array accumulation. Their
process() computes campaign-wide statistics over the same columns, so a
two-pass rewrite there is a larger change with no consumer today — psfex is
what every campaign runs.

BYTE-IDENTICAL OUTPUT, both ways in: the workflow's tar path and the module
runner's plain [path] path both give md5 f7caa1cf… on the fixture, unchanged
through both rewrites.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbnPCyzuDNTgkg715pHhar

The rule read every tile's catalogue on every run, because a DAG output must
be a function of its input set and rebuilding is the simple way to guarantee
that. At DR6 scale it is also ~800 GB of IO to add one 35 MB tile.

It now brings the file INTO AGREEMENT with the campaign: a tile with no
dataset is added, a dataset whose tile has left the campaign is deleted, a
dataset whose source catalogue CHANGED is re-read, and one that agrees with
its source is left alone, unread. Each dataset records its source's size and
mtime as attributes, and a mismatch is what changed means — which is also what
keeps the file from drifting from its inputs the way an append-only tool does.
create_final_cat.py's own process() implements only the append-only half of
this, skipping any tile already present whatever the file on disk now says.

WHAT IS AND IS NOT A FUNCTION OF THE INPUT SET, since this is the guarantee
being traded. The file's CONTENT is: the same tiles with the same catalogues
give the same datasets, the same columns and the same n_tiles, whether they
arrived at once or one campaign at a time. Its BYTE LAYOUT is not, because
hdf5 lays a group out in the order things were added. That is the price of not
re-reading the campaign.

UNTOUCHED ON A NO-OP, which is stronger than the byte comparison it replaces
and cheaper to establish: reconciling is PLANNED against a read-only open, and
an empty plan never opens the file for writing, so its mtime cannot move. A
non-empty plan is carried out on a copy which is then moved into place, so a
crash mid-merge leaves the old catalogue intact.

VERIFIED on a three-tile fixture: build (3 added), no-op (unchanged, mtime
identical to the nanosecond), append one tile WHILE AN EXISTING TILE'S
CATALOGUE IS UNREADABLE — chmod 000, which succeeds and reports 1 added, so
the existing tiles were demonstrably not read — rewrite of one catalogue
(1 refreshed), and dropping two tiles from the list (2 removed, datasets gone,
n_tiles 1). A from-scratch build of the same set is byte-stable across reruns.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbnPCyzuDNTgkg715pHhar

OPTIONAL COLUMNS WERE DECIDED ONCE FOR THE WHOLE MERGE. MergeStarCatPSFEX read
MAG/SNR/ACCEPTED out of the first catalogue's dtype and applied that verdict to
every file behind it, so a merge over a mix of ordinary and pix2wcs-converted
catalogues was wrong in both directions: ordinary first raised KeyError on the
first converted file, and converted first SILENTLY ZEROED the real values of
every ordinary file. The dtype now comes from any file that carries the column
and pass 2 asks each file for its own schema, so only the files that actually
lack a column are zero-filled. Both orderings verified on fixtures.

A FAILED psfex_interp COULD GET A GREEN MANIFEST, and clean_exposure takes that
manifest as its go-ahead to delete the store. With the default retention list,
an exposure whose interpolation failed but whose PSFEx model landed had a
non-empty match set, so the pack succeeded and the stars went with the store —
unrecoverable short of rebuilding the chain from VOS. psf_validation is not
optional: nothing matching it now fails the job, while the store is still on
disk. Retention products that match nothing stay warnings.

SHRINKING THE KEEP LIST DELETED PRODUCTS FROM /project. The list rides on
`params`, so editing it reruns the pack — which rewrote the tar without what
had been dropped, on the backed-up filesystem, with the scratch store it came
from usually already reclaimed. RETENTION IS NOW ADDITIVE: an existing tar is a
FLOOR, its members carried into the new one whatever the current list says, so
a config change can only ever add. Removing a product is a deliberate act on
products_dir, not a config edit. Verified: pack with [psf_model], rerun with an
empty list, the .psf is still there under its own product name and the tar is
byte-identical.

THE COLUMN SET REACHED NO RERUN TRIGGER. final_cat_merge's reconcile keyed
staleness on each source catalogue's size and mtime, and the column set is not
a source catalogue: final_cat.param arrives through `params`, and the hash
covered workflow/scripts/ only, not scripts/python/create_final_cat.py. So this
PR's own edit to final_cat.param would have left every dataset in an existing
hdf5 written to the old schema with nothing to notice. The file now carries a
digest of the resolved column list on its root and refreshes every tile when it
moves, and MERGE_FINAL_HASH covers all three files the rule's behaviour comes
from. Verified: build, edit the parameter file, rerun -> 3 refreshed.

STAR_CAT_MERGE'S MEMORY WAS SIZED ON THE TAR, which holds whatever the campaign
retains, while the merge reads the psf_validation members alone. Measured on a
fixture with a 3 MB PSF model kept: the tar is 92x the members it will read,
and the default retention is 2.4x. It also jumped discontinuously as exposures
were packed. The manifests record the product each member came from — exactly
so this is answerable without opening a tar — so the sizing sums those members.

NO REQUEST WAS CAPPED. A mem_mb above the partition maximum is a job SLURM
never schedules and snakemake never diagnoses: it sits PENDING while the
campaign looks alive. Both merge formulas grow with the campaign, so at some
size they cross it. `max_mem_mb:` (default 750000, for Nibi's 766 GB standard
node) caps both, with a parse-time warning naming the rule that was capped.

copy_data ORDERED ITS OUTPUT BY THE SOURCE CATALOGUE, which made the merged
dtype a property of the catalogue rather than of the parameter file: the
ordered dedup added to read_param_file had no effect, and two tiles written by
different ShapePipe versions landed in one group with two different structured
dtypes, which np.concatenate refuses. It orders by param_list now. Verified on
two catalogues with reversed column orders and an extra column: one dtype,
concatenate works. The fixture hdf5 md5 moves with the column order,
d2882294… -> 43ff946d….

RECONCILE LEAKED SPACE. It copied the file and deleted datasets in place, and
HDF5 never reclaims that, so every refresh of a tile grew the file by that
tile. A plan that removes or refreshes anything now builds the tmp fresh,
moving the datasets it keeps across with h5py's own group copy — a
dataset-level copy that never reads a row into numpy — so the result is
compact; pure-append plans still copy and append. Verified: five successive
full refreshes leave the file the same size, and dropping a tile shrinks it.

Also: comments referring to the deleted parse-time keep-list gate are gone;
`-s add` is documented as what it is (accepted by create_final_cat.py's
validator, then falling through to the ordinary walk, so not a way to add one
tile by hand); and three latent issues are noted where they live rather than
fixed — hdu.columns.dtype ignoring TSCAL/TZERO (no validation_psf column is
scaled), MergeStarCatSetools rebinding its ellipticity accumulators so only the
last file's reach the output (pre-existing, setools is not wired to any
workflow path), and the .tmp a SIGKILL can orphan next to the catalogue.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbnPCyzuDNTgkg715pHhar

… the tile one

The campaign's two products behaved differently for no reason anyone chose.
The shear catalogue reconciled — an append read the appended tiles — while the
star catalogue was one flat FITS table that had to be restacked from every
exposure the campaign had ever seen to add one: ~40 GB of members at DR6 scale
to add ~2 MB, held in memory while it happened. Now they are the same thing.

<products_dir>/full_starcat_<campaign>.hdf5, one dataset per exposure at
exposures/<exp>, each holding that exposure's every CCD's rows with a CCD_NB
column, an n_exposures root attribute and the same column digest the tile side
carries. Named for the campaign exactly as the shear catalogue beside it is.

CCD_NB IS AN INT: it is parsed out of the member name, where it is always
digits, so a string buys nothing and costs 8 bytes a row against 4. DTYPES ARE
NATIVE: float32 stays float32, where the FITS writer widened every float column
to 1D, doubling both the file and the peak memory of the job that wrote it for
no information.

MEMORY IS NOW FLAT IN THE CAMPAIGN — one exposure at a time — so the rule is
sized on the largest exposure's members rather than the campaign's, and the
~240 GB a DR6-scale flat table would have wanted is simply not a number any
more. The Snakefile's sizing block keeps the measurements that got us here,
because they are the argument for the format.

THE RECONCILE MACHINERY IS NOW ONE MODULE, workflow/scripts/hdf5_reconcile.py,
used by both merges rather than duplicated: plan against a read-only open,
add/refresh/remove, refresh everything when the column digest moves, compact
rewrite when anything is removed or refreshed, untouched on a no-op. Writing it
twice would have been two chances to disagree about what an output owes its
inputs.

THE WORKFLOW NO LONGER CALLS MergeStarCat* AT ALL, so merge_star_cat.py drops
the shapepipe import and the psf-model switch, and the [fileobj, name] entry
shape those classes learned for it is REVERTED — with no caller it was upstream
surface with nothing behind it. What stays upstream is what fixes the module
runner's own problems: the two-pass allocation, and asking each file for its
own optional columns instead of deciding once for the merge. The runner path is
byte-identical to before all of it, md5 f7caa1cf… on the fixture.

VERIFIED on the fixtures: build (2 added); no-op (unchanged, mtime identical to
the nanosecond); append one exposure with the others' tars at chmod 000, which
succeeds and reports 1 added, so they were demonstrably not read; remove one
(1 removed, dataset gone, n_exposures 2). Every one of the 16 columns equals
the FITS version's values. The tile side's compaction sequence was re-run
against a fix this work exposed — the keep-what-changed path called
Dataset.copy, which does not exist, and only bites when a plan both rewrites
and keeps something.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbnPCyzuDNTgkg715pHhar

path_hash KILLED EVERY INVOCATION over a file one rule needs. It runs at
module level, so a snapshot without scripts/python/ raised FileNotFoundError
during the parse — taking `sp --unlock`, `sp report` and every dry run with
it, and taking them with a bare traceback rather than the diagnosis
merge_final_cat.py already carries for exactly this case, which the job could
never reach. The hash degrades to a sentinel and warns once; the parse
survives, the rule still exists, and the job prints the message written for
it. Verified both halves with the file moved aside.

THE STAR MERGE'S OPTIONAL COLUMNS HAD NO CANONICAL DTYPE. MAG/SNR/ACCEPTED
took the dtype of whichever file carried them, falling back to X's float32
when none did — so an exposure whose files are all pix2wcs-converted got a
float32 ACCEPTED while its neighbours got int32, and datasets under exposures/
differed in dtype. np.concatenate refuses that, and no digest can repair it
because nothing about the schema CHANGED. The three are pinned (int32,
float32, float32) and cast. Verified on two exposures, one carrying them and
one not: identical dtypes, concatenate works.

A MISLABELED MANIFEST ABORTED THE CAMPAIGN. is_member() accepts a member by
product label OR by file name — right for "did this exposure keep the
product" — but read_exposure() selects by name alone, so an entry labelled
psf_validation whose name did not match put the exposure in the merge and then
killed the whole job when the tar held nothing selectable. Membership is now
the name on both sides, with one line saying an entry was labelled and skipped.

RENAMING `campaign:` WOULD HAVE HALF-UPDATED THE FILE. The tile hdf5 carries
the campaign in its GROUP, so a rename pointed the rule at a new group inside
the same file: a second group beside the first, the first frozen and stale,
and n_tiles describing one of them. One file is one campaign — apply refuses
and names what is already there.

APPEND IS CHEAP IN READS, NOT IN WRITES, and the docstrings said otherwise.
The existing file is copied so the result can be moved into place atomically:
one pass over it and, briefly, twice its size on disk. Corrected, and apply
now refuses when the filesystem cannot hold it rather than filling /project
and leaving a truncated tmp beside a catalogue people trust.

A CORRUPT TAR RAISED A RAW ReadError, on both sides. persist_exp now refuses
to write a new tar and says the old one is untouched and may hold products
nothing else has; merge_star_cat names the tar and says not to delete it.

THE 16-COLUMN SCHEMA IS DEFINED TWICE and nothing held the two together.
MergeStarCatPSFEX writes the flat FITS table the module runner emits;
merge_star_cat.py writes the hdf5. Separate implementations are right — only
one of them reads tars, keeps native dtypes and reconciles — but a column
added to one writer would simply be missing from the other's product, found by
whoever next computed rho statistics from the wrong one.
tests/unit/test_star_cat_columns.py asserts the names and their order agree;
verified passing, and verified failing when one list is changed.

Also noted where it lives: adding a retention product re-packs the tar and
moves its mtime, so the star merge refreshes those exposures although their
validation members are byte-for-byte unchanged — seconds per exposure against
per-member bookkeeping on every exposure, which is not a trade worth making.

Stale docs updated to the hdf5 product: the Snakefile's merges header, the
README's star_cat_merge paragraph and scripts list (the hdf5 paragraph written
last round never landed — its edit script aborted before writing), and
config.yaml's psf_validation block. The README now says there are two writers
and names the test that keeps their schema together.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QbnPCyzuDNTgkg715pHhar